Method and apparatus of processing a signal from an event-based sensor

ABSTRACT

For processing a signal from an event-based sensor having an array of sensing elements facing a scene, the method comprises: receiving the signal including, for each sensing element, successive events originating from said sensing element depending on variations of incident light from the scene; analyzing the signal to detect a frequency pattern in a light profile sensed by at least one sensing element; and extracting information from the scene in response to detection of the frequency pattern.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the U.S. national phase of the International PatentApplication No. PCT/EP2019/051919, filed Jan. 25, 2019, which claims thebenefit of European Patent Application No. 18305063.2, filed Jan. 26,2018, and European Patent Application No. 18191469.8, filed Aug. 29,2018, the contents of all are incorporated herein by reference.

FIELD

The present disclosure relates to machine vision, especially to a methodand a system of processing a signal from an event-based sensor.

BACKGROUND

Machine vision is a field that includes methods for acquiring,processing, analyzing and understanding images for use in wide type ofapplications such as for example security applications (e.g.,surveillance, intrusion detection, object detection, facial recognition,etc.), environmental-use applications (e.g., lighting control), objectdetection and tracking applications, automatic inspection, processcontrol, and robot guidance etc. Therefore, machine vision can beintegrated with many different systems.

In order to extract information from a scene, it is common to usesequences of images (frames) delivered by image sensors (i.e. videocameras). Such frame-based image sensors acquire the visual informationfrom the scene either sequentially for each sensor element (pixel) oreach pixel line/column or, in various patterns, pixel parallel, butalways time-sampled at some limited frame rate.

However, in the scene to be analyzed, there might be light sources suchas for example fluorescent lights, compact fluorescents, light emittingdiode (LED) lights, television screens, movie projectors, or computerscreens that are flickering, either because of the way they are powered(for instance, fluorescent lights flicker at twice the AC powerfrequency) or because of they are driven with periodic pulsed signals atfrequencies indiscernible for the human eye, to control their lightoutput or power consumption (like, e.g., many LED light sources).

This is a problem for frame-based video capture where different andunrelated frame acquisition frequencies and light frequency patternsinterfere, as well as different light duty cycles and exposure times. Asa consequence, such frequency patterns can make a light source appeardifferent in different frames, e.g. being ON in some frames andappearing OFF in others.

This problem is more serious for the machine vision applications, whichare based on reliable capture of information from or at least inpresence of artificial light sources. For example, in automotiveapplications (driver assistance, autonomous driving), it is useful tocapture the status of traffic lights (which oftentimes use LED sources),road signs, break and tail lights of other cars, etc. at all timeswithout interruptions or uncertainties. Automotive image and videocapture is particularly affected. Due to high dynamic rangerequirements, often multi-exposure image sensors that execute multipleexposures at different exposure times for each final image frame areused.

Furthermore, due to the combination of nonuniform light source frequencypatterns and complex image information sampling processes (withdifferent shutter time combinations), for frame-based sensors, the stateof the light source can evolve through the frames in unexpected ways,causing artefacts like seeing only half of the digits of a speed limitsign or receiving wrong color information from a traffic light. Greateffort is put on the development of frame-based image sensors that areable to suppress the described effects of light frequency patterns andmake the sensor output appear as if it would look at an unmodulatedstatic light source. However, due to the characteristics of frame-basedsensors, it is generally difficult to capture accurate information,especially from modulated light.

A new event-based machine vision paradigm is developed. Instead ofcapturing an image with all the pixels at the same time and trying tocompensate for the artifacts and suppress information, an event-basedimage sensor acquires the temporal luminance profile sensed by eachpixel from the scene viewed by the sensor. Event-based sensors cancapture fast changing information in a visual scene due to the hightemporal resolution of the information individually acquired by thepixels. More details of such an event-based machine vision technology isdiscussed in the article Posch, C., Serrano-Gotarredona, T.,Linares-Barranco, B., & Delbruck, T. (2014), “Retinomorphic event-basedvision sensors: bioinspired cameras with spiking output”, Proceedings ofthe IEEE, 102(10), 1470-1484.

Accordingly, an object is to provide a new approach of capturinginformation from scenes viewed by event-based sensors.

SUMMARY

A method of processing a signal from an event-based sensor is provided.The event-based sensor having an array of sensing elements facing ascene, the method comprises: receiving the signal including, for eachsensing element, successive events originating from said sensing elementdepending on variations of incident light from the scene; detecting afrequency pattern in a light profile sensed by at least one sensingelement by analyzing the said signal; and extracting information fromthe scene in response to detection of the frequency pattern.

A frequency pattern, which may be perceived as flicker, is noteliminated but captured from the output of the event-based sensor inorder to be exploited to extract some useful information. The extractedinformation may be encoded beforehand, or natively present in the sceneobserved by the array of sensing elements. The method takes advantage offlickering effects, rather than dismissing them as undesirable.

In a method according to an embodiment, the event-based sensor has anarray of sensing elements facing a scene, and each event may comprise atime of occurrence and a luminance attribute corresponding to avariation of incident light. The method can be used to analyze thesignal through steps including:

extracting from the signal successive bursts of events originating fromat least one sensing element, each burst consisting of events havingluminance attributes that correspond to variations of incident light ofa same direction; and

detecting a frequency pattern based on timing of the successive bursts.

Such a method of analyzing the signal has an application to extractinformation from the scene, using the detected frequency pattern(s), asmentioned above. However, other applications exist, for example forcancelling frequency-modulated components, or flickering effects, fromthe signal for further processing or display.

A burst may be extracted from the signal by counting a preset number ofevents originating from a same sensing element, having luminanceattributes that correspond to variations of incident light of a samedirection.

In an embodiment, analyzing the signal further comprises filteringevents received in the signal, the successive bursts of events beingextracted from the signal after filtering. The filtering comprises, uponreceiving a new event originating from a sensing element of theevent-based sensor:

discarding the new event if the time of occurrence of the new event ismore than a memorized time of occurrence of a preceding event from thesame sensing element plus a time threshold;

discarding the new event if the luminance attribute of the new event anda memorized luminance attribute of the preceding event from the samesensing element correspond to variations of incident light of oppositesign; and

memorizing the time of occurrence and the luminance attribute of the newevent for filtering a next event from the same sensing element.

Extracting a burst of events having a polarity corresponding to adirection of variation of incident light may comprise, for a group of Nsensing elements, where N is an integer equal to or greater than 1,incrementing an event counter when the filtered signal has a new eventoriginating from a sensing element of the group with a luminanceattribute that corresponds to the polarity of the burst. The burst canthen be extracted when the event counter reaches a threshold. Extractingthe burst of events having the polarity may further comprise, when theevent counter reaches the threshold, changing the polarity forextraction of a next burst of events. In some implementations,extracting the burst of events comprises resetting the event counterwhen the signal has a new event originating from a sensing element ofthe group with a luminance attribute that does not correspond to thepolarity of the burst. The event counter may also be reset if the eventcounter does not reach the threshold within a given time slice.

In an embodiment, detecting the frequency pattern comprises determiningfirst time intervals between consecutive bursts consisting of eventshaving luminance attributes that correspond to variations of incidentlight of a same direction; and performing statistical analysis of thefirst time intervals to decide whether the successive bursts indicate afrequency pattern.

When it is decided that the successive bursts indicate a frequencypattern, detecting the frequency pattern may further comprise estimatinga frequency of the frequency pattern based on the first time intervals.It is also possible to estimate a duty cycle of the frequency patternbased on second time intervals determined between consecutive burstsconsisting of events having luminance attributes that correspond tovariations of incident light of opposite directions.

Some embodiments of the method may use of a kind of spatial filtering ofthe signal from the event-based sensor after detecting the frequencypattern. Such an embodiment comprises determining a spatial region ofthe scene where the same frequency and/or duty cycle is detected, andignoring the spatial region if a size of the spatial region exceeds adefined size.

In an application of the method to the transmission of information,extracting information may comprise demodulating frequency patternsdetected in the scene. For example, the frequency patterns may bedemodulated by referring to On-Off keying (OOK), or Manchester coding.

The information extracted from the scene may output with locationinformation relating to an address, in the array, of at least onesensing element where the frequency pattern is detected.

Another aspect of an embodiment relates to an apparatus for extractinginformation from a scene, comprising: an array of sensing elements to beplaced in front of the scene; and at least one processor coupled withthe array of sensing elements and configured to perform a method asmentioned above.

Yet another aspect of an embodiment relates to a computer programcomprising program code to be executed by a processor coupled with anarray of sensing elements, the program code being adapted to performanceof a method as mentioned above when executed by the processor.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features and advantages of the present disclosure will appear inthe description hereinafter, in reference to the appended drawings,where:

FIG. 1 is a block diagram of a device according to an embodiment;

FIG. 2 shows an example of a light profile received by an asynchronoussensor and a signal generated by the asynchronous sensor in response tothe light profile;

FIG. 3 is a flowchart of an illustrative process for detecting bursts ofevents; and

FIGS. 4A-B are flowcharts of an alternative process for detecting burstsof events; and

FIG. 5 is a flowchart illustrating a method according to an embodiment.

DETAILED DESCRIPTION

The apparatus shown in FIG. 1 comprises an event-based asynchronousvision sensor 10 placed facing a scene and receiving the light flow ofthe scene through optics for acquisition 15 comprising one or severallenses. The sensor 10 is placed in the image plane of the optics foracquisition 15. It comprises an array of sensing elements, such asphotosensitive elements, organized into a matrix of pixels. Each sensingelement corresponding to a pixel produces successive events depending onvariations of light in the scene.

A processor 12 processes the information originating from the sensor 10,i.e. the sequences of events received asynchronously from the variouspixels, in order to extract therefrom information included in the scene.It can be implemented by programming using a suitable programminglanguage. A hardware implementation of the processor 12 usingspecialized logic circuits (ASIC, FPGA, . . . ) is also possible.

For each sensing element, the sensor 10 generates an event-based signalsequence depending on the variations of light received by the sensingelement from the scene that appears in the field of vision of thesensor.

The asynchronous sensor 10 carries out an acquisition to output a signalwhich, for each pixel, may be in the form of a succession of instantst_(k) (k=0, 1, 2, . . . ) at which an activation threshold Q is reached.Each time this luminance increases by a quantity equal to the activationthreshold Q starting from what it was in time t_(k), a new instantt_(k+1) is identified and a spike is emitted at this instant t_(k+1).Symmetrically, each time that the luminance observed by the sensingelement decreases by the quantity Q starting from what it was in timet_(k), a new instant t_(k+1) is identified and a spike is emitted atthis instant t_(k+1). The signal sequence for the sensing elementincludes a succession of spikes positioned over time at instants t_(k)depending on the light profile for the sensing element. The output ofthe sensor 10 is then in the form of an address-event representation(AER). In addition, the signal sequence typically includes a luminanceattribute corresponding to a variation of incident light.

The activation threshold Q can be fixed, or can be adapted as a functionof the luminance. For example, the threshold can be compared to thevariations in the logarithm of the luminance for generating events whenexceeded.

By way of example, the sensor 10 can be a dynamic vision sensor (DVS) ofthe type described in “A 128×128 120 dB 15 μs Latency AsynchronousTemporal Contrast Vision Sensor”, P. Lichtsteiner, et al., IEEE Journalof Solid-State Circuits, Vol. 43, No. 2, February 2008, pp. 566-576, orin patent application US 2008/0135731 A1. The dynamics of a retina(minimum duration between the action potentials) can be approached witha DVS of this type. The dynamic behavior surpasses that of aconventional video camera that has a realistic sampling frequency. Whena DVS is used as the event-based sensor 10, data pertaining to an eventoriginating from a sensing element include the address of the sensingelement, a time of occurrence of the event and a luminance attributecorresponding to a polarity of the event, e.g. +1 if the luminanceincreases and −1 if the luminance decreases.

Another example of an asynchronous sensor 10 that can be usedadvantageously in the context of an embodiment is the asynchronoustime-based image sensor (ATIS) of which a description is given in thearticle “A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor WithLossless Pixel-Level Video Compression and Time-Domain CDS”, C. Posch,et al., IEEE Journal of Solid-State Circuits, Vol. 46, No. 1, January2011, pp. 259-275. When an ATIS is used as the event-based sensor 10,data pertaining to an event originating from a sensing element includethe address of the sensing element, a time of occurrence of the eventand a luminance attribute corresponding to an estimated value of theabsolute luminance. The luminance attribute in the ATIS also includesthe temporal contrast event polarity.

According to special embodiment, the present disclosure aims atdetecting frequency patterns in the scene viewed by the event-basedsensor 10.

By way of illustration, FIG. 2 shows, with arbitrary scales, an exampleof light profile P which may be seen by a sensing element when there issome frequency pattern in the luminance, and an example of signalgenerated by a sensor of the DVS type for such sensing element.

The light profile P seen by the sensing element includes a frequencypattern which, in the simplified example shown for the explanation, hasalternating rising and falling edges with a periodicity of T, i.e. afrequency of f=1/T. At t0, when the luminance increases by a quantityequal to the activation threshold Q, an event is generated by thesensing element. The event has a time of occurrence t0 and a luminanceattribute, i.e. polarity in the case of a DVS (level +1 at t0 in FIG.2). With the increase of the luminance in the rising edge, subsequentevents with the same positive polarity are generated each time theluminance further increases by the activation threshold. These eventsform a burst denoted as B0 in FIG. 2. When the falling edge starts, theluminance decreases and gives rise to another burst B1 of events ofopposite polarity, i.e. negative polarity (level −1), from time t1.Further rising and falling edges in the luminance sensed by the sensingelement produce other bursts of positive spikes, such as B2 from timet2, and other bursts of negative spikes, such as B3 from time t3.

The diagram in the lower part of FIG. 2 is shown in the illustrativecase of a DVS sensor.

FIG. 3 shows a flow chart of an illustrative process for detectingbursts of events which can be used in various implementations. Thefollowing notations are used in the figure:

-   -   A: address of a current event, identifying its location in the        matrix of pixels of the sensor 10;    -   t: time of occurrence of the current event;    -   p: polarity of the current event, e.g. p=+1 if the sensed        luminance is increasing and p=−1 if the sensed luminance is        decreasing;    -   t′(A): time of occurrence of the latest event indicated by the        sensing element at address A;    -   p′(A): polarity of the latest event indicated by the sensing        element at address A. p′(A) also denotes the polarity of a burst        of events which is being extracted; δt_(max): a preset time        threshold;    -   C(A): an event counter for the pixel address A.

The process shown in FIG. 3 includes a filtering operation 20 which iscarried out upon receipt of each event from the array of sensingelements, followed by a burst extraction stage 30. The processor 12waits for new events (step 21) and upon receipt of an event withparameters A, t, p at step 22, starts the filtering operation 20.

The filtering operation 20 includes steps 24-27 performed pixel-wise.First, the polarity p of the received event is compared with that p′(A)of the preceding event from the same pixel. If p≠p′(A) (test 24), theevent may be spurious (for the purpose of detecting frequency patterns)or denote the onset of a rising or falling edge of the luminanceprofile. The event for which p≠p′(A) is discarded and, before returningto step 21 of waiting for the next event from the sensor 10, theparameters p′(A), t′(A) and C(A) are reset to take the values p, t and1, respectively, at step 25. If p=p′(A) at test 24, another test 26 isperformed to compare the times of occurrence t, t′(A) of the current andpreceding events with respect to the time threshold δt_(max). If t−t′(A)δt_(max), it is considered that the time elapsed since the precedingevent is too long for both events to be part of a rising or falling edgeof the luminance profile. In this case, the filtering operation 20proceeds to the above-mentioned resetting step 25. Otherwise, ift−t′(A)<δt_(max), the current event is kept in the stream of events and,optionally, its time of occurrence t is memorized as the parameter t′(A)at step 27 in view of processing the next event at the same address A.

In the embodiment illustrated in FIG. 3, the burst extraction stage 30is also performed pixel-wise. After step 27, the event which has justbeen kept is taken into account by incrementing the event counter C(A)for the pixel address A (step 31).

Then, the event counter C(A) is compared with a threshold C_(min) atstep 32. While C(A)<C_(min), no burst is extracted, and the processor 12returns to step 21 of waiting for the next event. When C(A)=C_(min) atstep 32, it is determined that the signal includes a burst at address A,which is extracted at step 33 by noting its address A, time t andpolarity p. In this preferred example where step 27 is used, timing ofthe extracted burst is denoted by the time of occurrence t of itsC_(min)-th event (if) occurring with time intervals that do not exceedδt_(max). It is noted that the parameters of the burst extracted at step33 are produced in an event-based manner, using AER.

The preset parameters δt_(max) and C_(min) are chosen depending onfactors such as the dynamics of the event-based sensor 10 which is used,and requirements of the application run by the processor 12 includingfrequency ranges to be detected. By way of example and withoutlimitation, δt_(max) may be in a range of 0.1 to 50000 μs and C_(min)may be between 2 and 10.

After extracting a burst at step 33, the process returns to the waitingstep 21. Another option (not shown) is that, after extracting the burstat step 33, the event counter C(A) is reset and the polarity p′(A) ischanged (to −p) for subsequent extraction of a next burst of events,before returning to the waiting step 21.

In an alternative embodiment, the filtering stage 20 is not used beforethe burst extraction stage 30. In this case, the events are processed instage 30 in the case where they have the same polarity as the currentburst, namely p=p′(A).

FIG. 4, formed by combining FIGS. 4A and 4B, illustrates an alternativeembodiment of the process for detecting bursts of events. The notationsare the same as in FIG. 3. However, there is an additional parameterΔt_(max) representing a time slice in which consecutive events of a samepolarity must fall for a burst to be extracted. The variable t″(r)indicates the time of occurrence of the first event of a current burstin a region r.

In the example of FIGS. 4A-B, the bursts are detected over regions ofthe array of sensing elements. Each region is made of a group of Nsensing elements. The number N is an integer at least equal to 1, whichmay be different from one region to another. A number of regions,typically made of adjacent sensing elements, are defined over the array.These regions may be overlapping, which means they may share somesensing elements and thus a given spike may belong to several regions.Such arrangement of regions is adapted to form a spatial pooling of datafor filtering out noise.

FIG. 4A is essentially the same as the upper part of FIG. 3. Theanalysis of the signal comprises a stage 20 of filtering events as theyare received in the signal provided by the array of sensing elements.The filtering 20 may be the same as in FIG. 3, except that it is notnecessary to initialize a pixel-wise event counter at step 25. Instead,region-wise event counters C(r) are used. For each region index r, theevent counter C(r) is initialized to 0 at the outset.

The burst extraction stage 40 of FIG. 4B is performed at the regionlevel, following step 27 of FIG. 4A. If the address A for which an eventis retained at step 27 belongs to a number m(A) of regions R_(i)(A)(1≤i≤m(A)) defined over the array, a loop is performed to increment theevent counters of these regions and decide whether bursts has occurredin these regions. After step 27, a first region is selected at step 401,by taking i=1, and its region index r=R_(i)(A) is derived at step 402,and then the event is taken into account by incrementing the eventcounter C(r) for this first region index (step 41).

Then the event counter C(r) is compared with the threshold Gin at step42. The value of the threshold C_(min) may be dependent on the size ofregion R_(i)(A), for example proportional to the number of sensingelements of which region R_(i)(A) is made. While C(r)<C_(min) at step42, no burst is extracted. In this case, the variable t″(r) receives thevalue of the time of occurrence t of the current event at step 44 ifC(r)=1 at test 43. After step 44, or when C(r)>1 at test 43. The regionindex r is compared with the number m(A) of regions to which address Abelongs at step 47. When r<m(A) at step 47, i is incremented by 1 atstep 403 for taking the current event into account in the next regionR_(i+1)(A) from step 402. When r=m(A) at step 47, the processor 12returns to step 21 of waiting for the next event.

When C(r)=C_(min) at step 42, it is checked at test 45 whether theC_(min) consecutive events having the same polarity p fall within thepreset time slice Δt_(max). If t−t″(r) Δt_(max) at test 45, no burst isextracted because it is considered that a burst cannot last so long, andthen C(r) and t″(r) are reset to take the values 1 and t at step 48.When t−t″(r)<Δt_(max) at test 45, it is determined that the signalincludes a burst in region r. The burst is extracted at step 46 bynoting its region index r, time t″(r) and polarity p. In this example,timing of the extracted burst is denoted by the time of occurrence t″(r)of its first event. It is noted that the parameters of the burstextracted at step 46 are produced in an event-based manner, using AER.Afterwards, the region index r is compared with the number m(A) ofregions to which address A belongs at the above-described step 47.

The preset parameter Δt_(max) is chosen depending on factors such as thedynamics of the event-based sensor 10 which is used, and requirements ofthe application run by the processor 12 including frequency ranges to bedetected, as well as the values of δt_(max) and C_(min). By way ofexample and without limitation, Δt_(max) may be in a range of 100 μs to100 ms.

Once a number of successive bursts have been extracted for a pixel A ora region r of N pixels using a process as illustrated in FIG. 3 or 4,for example, the timing of those successive bursts can be analyzed inview of detecting a frequency pattern.

In FIG. 5, step 60 denotes the extraction of successive bursts which canbe performed using a process as shown in one of FIGS. 3 and 4 on thesignal received at step 50 from the event-based sensor 10, and step 70corresponds to the analysis of the successive bursts to detect frequencypatterns. Steps 60 and 70 together form the analysis step 80 applied tothe received signal to detect frequency patterns therein. Once afrequency pattern is extracted at step 70, it can be exploited toextract some information available in the scene at step 100. As shown inFIG. 5, an optional spatial filtering 90 can be applied between theanalysis step 80 and the step 100 of extracting information from thescene.

By way of example, when implementing the analysis step 70, somestatistical analysis of time intervals between consecutive burstsconsisting of events having the same polarity is performed in order todetect a frequency pattern.

For example, a buffer can be provided to store the times of successivebursts extracted in a given region (consisting of one or more sensingelements) and having a given polarity. For each polarity p=±1, the timeintervals Δt₁ between consecutive bursts is noted, and their values forboth polarities can be analyzed to estimate a frequency value f=1/T (seeFIG. 2), for example by averaging. Such frequency value may be validatedto decide that a frequency pattern is present if the standard deviationof the values Δt₁ is within a fraction (e.g. 1 to 10%) of their meanvalue after a number M of bursts (e.g. 3≤M≤20). Alternatively, thevalues of the time intervals Δt₁ for both polarities are filtered usingaveraging or median filtering, and it is decided that a frequencypattern is present at frequency f=1/T if a buffer of the filtered valuescontains more than M identical values T.

It is also possible, if a frequency pattern is detected, to estimate aduty cycle D of that frequency pattern as a further parameter describingthe frequency pattern. To do this, other time intervals Δt₂ areexamined. Each of the time intervals Δt₂ is between a burst of polarityp=+1 (indicating a rising edge of the light profile) and a subsequentburst of polarity p=−1 (indicating a falling edge of the light profile).The mean value, or the majority value of the time intervals over theduration for detecting the burst, provides an estimate of the timeoffset ΔT shown in FIG. 2. The duty cycle D can then be estimated asD=ΔT/T.

Once a frequency pattern is detected and related parameters such as f, Dare determined, the processor 12 has information about the location ofthe frequency pattern in the scene viewed by the event-based sensor 10.Such location information can be in the form of the address A of thesensing element where the pattern is detected, or in the form of anindex region R. Such location information is relevant in machine visionapplications, in order to determine where a source of modulated light islocated.

If a plurality of frequency patterns is detected with a same frequency fin the scene viewed by the event-based sensor, it is also possible toderive a phase shift between those patterns, by analyzing the timedifference between their rising edges and or their falling edges.

Optionally, after step 70 shown in FIG. 5, a further spatial filteringstep 90 can be executed. In particular, when a large spatial region ofthe sensor is showing the same blinking pattern, i.e. same frequencyand/or same duty cycle, it can be considered that such frequency patternis not suitable for the extraction of information from the scene. Forexample, such large frequency patterns may come from a reflectivesurface such as a wall reflecting/diffusing light from some lightsource. In this case, the events from those sensing elements, such aspixels, are discarded in step 90, so as to be ignored in the informationextraction step 100. On the contrary, when the spatial size of afrequency pattern is small, these sensing elements are considered facinga blinking light source. Therefore, the actual position of a lightsource can be determined. This may be useful, for example, forrecognizing LED braking lights and traffic lights in automotiveapplications.

The information extraction step 100 is executed so as to extractinformation from frequency patterns in the scene by analyzing ordemodulating the frequency and/or duty cycle of a modulated light.

Therefore, the method can sample the signal to find frequency patternsand extract information about the light source such as the exact timesof transitions, modulation frequency, phase, duty cycle, etc. Inaddition, due to the autonomous functioning of the individual sensingelements, the information can be read out from the array in the form ofsensing element addresses or indexes, and a detected frequency patternassociated with a light source can also facilitate determining otherinformation about the light source in the scene.

In an alternative embodiment, the extracted information can be used todetect an object in a scene by simply recognizing the frequency patternof the light source, such as a nominal frequency and/or duty cycle ofsome light source, such as traffic light, braking lights and other lightsources.

Alternatively, in another embodiment, a light source can be provided totransmit some information, such as an ID, encoded in basiccharacteristics of the light modulation, such as frequency and/or dutycycle. In addition, the light source may also contain more sophisticatedmodulation of light with encoded information, so as to transmit a largeramount of information. Hence, the method disclosed herein makes itpossible to use an event-based sensor 10 as a receiver for Visible LightCommunication (VLC) or LiFi (see, e.g., the IEEE 802.15.7 standard).

To demodulate information transmitted using VLC, frequency patternsproduced by the modulated light are extracted by the above-mentionedstep 80. The information is then demodulated and extracted from thesefrequency patterns.

The information transmitted through VLC can be encoded or modulated bymethods such as On-Off-keying (OOK) and Manchester coding. It ispossible to use a set of different frequencies and/or duty cycles tocode for different symbols, which enables an increase of bandwidth.

Due to the independent and autonomous operation of the pixels thatrespond to the light transitions, information can independently beacquired from multiple light sources in the scene and decodedseparately. In that sense, an event-based sensor can be seen as amulti-channel VLC receiver where all the communicating light sources inthe field of view of the sensor, such as lighting on the side of theroad and VLC-modulated lights of the different cars, can be seen at thesame time and independently by the sensor. The sensor does not need tobe steered towards each light source provided that the sources are inthe field of view. They can then be independently detected anddemodulated.

In addition to visible light, the light can also be infrared ornear-infrared light.

The processor 12 shown in FIG. 1 is embodied separately from theevent-based sensor 10. Alternative architectures are possible. Inparticular, part or all of the processing illustrated in FIG. 5 can beperformed in the same component as the event-based sensor 10, in anon-chip architecture. It is also possible to split the tasks betweenmore than one processors operating sequentially or in parallel.

The above-described method may be implemented using program instructionsrecorded in non-transitory computer-readable media to implement variousoperations which may be performed by a computer. The media may alsoinclude, alone or in combination with the program instructions, datafiles, data structures, and the like. The program instructions recordedon the media may be those specially designed and constructed for thepurposes of the illustrative embodiments, or they may be of thewell-known kind and available to those having skill in the computersoftware arts. Examples of non-transitory computer-readable mediainclude magnetic media such as hard disks, floppy disks, and magnetictape; optical media such as CD ROM discs and DVDs; magneto-optical mediasuch as optical discs; and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory (ROM), random access memory (RAM), flash memory, and the like.Examples of program instructions include both machine code, such as codeproduced by a compiler, and files containing higher level code that maybe executed by the computer using an interpreter. The described hardwaredevices may be configured to act as one.

In an example of application, the apparatus including the event-basedsensor 10 and the processor(s) 12 is used in an automobile to detecttraffic information, such as traffic light, moving objects, etc., so asto provide traffic information to autonomous driving system.

For example, according to a special embodiment, given the pixels and thetimes on which a blinking pattern has been detected, pixels are groupedtogether according to spatial vicinity and frequency. More precisely,when the same frequency is detected for groups of pixels which are closespatially and at times within a small interval, these pixels areconsidered as belonging to a consistent region. Such a region in a sceneis then classified as belonging to a LED light source such as trafficlights, turning lights/braking lights form a car, street lights etc.This classification is done according to the size of the extractedregion, its frequency, the position in the array of pixels.

According to this embodiment, a car (or a vehicle) containing multipleblinking patterns can be detected more reliably by detecting pairs ofsuch regions: for each detected region, another region of consistentsize, phase and frequency is determined together with the first onewhich also lies on a horizontal line from the first region and which isnot too far from it.

The relative movement and the distance from the event-based sensor of, avehicle detected with this method can be estimated by tracking over timethe pixel distance between the two regions representing two sources ofblinking lights.

Assuming the separation between the two blinking sources is fixed, adistance Ds between the two regions in the scene is correlated with adistance Dv between the vehicle and the event-based sensor.

Assuming the separation between the two blinking sources is fixed, therate of change of the distance Ds between the two regions in the sceneis correlated to the relative motion. In particular the distance Dsgetting smaller means the vehicle is getting further and the distance Dsgetting bigger means the vehicle is getting closer. More precisely, timeof contact to the object can be estimated by the ratio between thedistance Dv and the derivative of the distance Dv (see for example:Negre, Amaury, et al. “Real-time time-to-collision from variation ofintrinsic scale.” Experimental Robotics. Springer, Berlin, Heidelberg,2008.)

In addition, assuming the height of the light compared to the ground isfixed, its vertical position in the scene is correlated with thedistance Dv between the vehicle and the camera, which is adapted to beused for single blinking light, such as single lights on motorbikes.

More generally, embodiments may be useful in a variety of devicesincluding, without limitation, autonomous vehicles, prosthetic devices,autonomous and robotic apparatus such as manufacturing robots, militaryor medical robotic devices.

In an other example of application, the apparatus including theevent-based sensor 10 and the processor(s) 12 is used to detect thepattern projected on objects or surfaces by a laser-light pointer,detecting a region of one or more pixels blinking at the laser frequencypattern.

Embodiments may further be applicable in a wide variety of stationaryand portable devices, such as, for example, smart phones, portablecommunication devices, notebook, netbook and tablet computers,surveillance systems, video-projectors and practically any othercomputerized device configured to process vision data.

Embodiments may be used in many applications including computer humaninteraction (e.g., recognition of gestures, voice, posture, face,pointing devices and/or other applications), controlling processes(e.g., an industrial robot, autonomous and other vehicles), followingmovements of a set of interest points or objects (e.g., vehicles orhumans) in the visual scene and with respect to the image plane,augmented reality applications, virtual reality applications, accesscontrol (e.g., opening a door based on a gesture, opening an access waybased on detection of an authorized person), detecting events (e.g., forvisual surveillance or people or animal), counting, tracking, etc.Myriads of other applications exist that will be recognized by those ofordinary skilled in the art given the present disclosure.

The embodiments described hereinabove are exemplary illustrations, andvarious modifications can be made to them without leaving the scope ofthe disclosure which stems from the annexed claims.

1: A method of processing a signal from an event-based sensor, theevent-based sensor having an array of sensing elements facing a scene,the method comprising: receiving the signal including, for each sensingelement, successive events originating from said sensing elementdepending on variations of incident light from the scene; analyzing thesignal to detect a frequency pattern in a light profile sensed by atleast one sensing element; and extracting information from the scene inresponse to detection of the frequency pattern. 2: The method of claim1, wherein the events each comprise a time of occurrence and a luminanceattribute corresponding to a variation of incident light, and whereinanalyzing the signal comprises: extracting from the signal successivebursts of events originating from at least one sensing element, eachburst consisting of events having luminance attributes that correspondto variations of incident light of a same direction; and detecting thefrequency pattern based on timing of the successive bursts. 3: Themethod of claim 2, wherein a burst is extracted from the signal bycounting a preset number of events originating from a same sensingelement, having luminance attributes that correspond to variations ofincident light of a same direction. 4: The method of claim 2, whereinanalyzing the signal further comprises filtering events received in thesignal, the successive bursts of events being extracted from the signalafter filtering, and wherein the filtering comprises, upon receivinganew event originating from a sensing element of the event-based sensor:discarding the new event if the time of occurrence of the new event ismore than a memorized time of occurrence of a preceding event from thesame sensing element plus a time threshold; discarding the new event ifthe luminance attribute of the new event and a memorized luminanceattribute of the preceding event from the same sensing elementcorrespond to variations of incident light of opposite sign; andmemorizing the time of occurrence and the luminance attribute of the newevent for filtering a next event from the same sensing element. 5: Themethod of claim 4, wherein extracting a burst of events having apolarity corresponding to a direction of variation of incident lightcomprises, for a group of N sensing elements, where N is an integerequal to or greater than 1: incrementing an event counter when thefiltered signal has a new event originating from a sensing element ofthe group with a luminance attribute that corresponds to the polarity ofthe burst, the burst being extracted when the event counter reaches athreshold. 6: The method of claim 5, wherein extracting the burst ofevents having the polarity further comprises, when the event counterreaches the threshold, changing the polarity for extraction of a nextburst of events. 7: The method of claim 5, wherein extracting the burstof events comprises resetting the event counter when the signal has anew event originating from a sensing element of the group with aluminance attribute that does not correspond to the polarity of theburst. 8: The method of claim 5, further comprising resetting the eventcounter if the event counter does not reach the threshold within a giventime slice. 9: The method of claim 2, wherein detecting the frequencypattern comprises: determining first time intervals between consecutivebursts consisting of events having luminance attributes that correspondto variations of incident light of a same direction; and performingstatistical analysis of the first time intervals to decide whether thesuccessive bursts indicate a frequency pattern. 10: The method of claim9, wherein detecting the frequency pattern further comprises, when it isdecided that the successive bursts indicate a frequency pattern,estimating a frequency of the frequency pattern based on the first timeintervals, and/or estimating a duty cycle of the frequency pattern basedon second time intervals determined between consecutive burstsconsisting of events having luminance attributes that correspond tovariations of incident light of opposite directions. 11: The method ofclaim 1, wherein further comprising, after detecting the frequencypattern: determining a spatial region of the scene where the samefrequency and/or duty cycle is detected; ignoring the spatial region ifa size of the spatial region exceeds a defined size. 12: The method ofclaim 11, further comprising, after determining the spatial region:detecting geometry dimension of the spatial region in the scene;calculating a distance or relative motion between a light sourcegenerating the spatial region and the event-based sensor in correlationwith the detected geometry dimension. 13: The method of claim 1, whereinextracting information comprises demodulating frequency patternsdetected in the scene. 14: The method of claim 1, wherein theinformation extracted from the scene is output with location informationrelating to an address, in the array, of at least one sensing elementwhere the frequency pattern is detected.
 15. (canceled) 16: Acomputer-readable medium having a computer code stored thereon forexecution by a processor coupled with an array of sensing elements, thecomputer code being adapted, when executed by the processor, to processa signal from an event-based sensor comprising the array of sensingelements facing a scene, wherein processing the signal comprises:receiving the signal including, for each sensing element, successiveevents originating from said sensing element depending on variations ofincident light from the scene; analyzing the signal to detect afrequency pattern in a light profile sensed by at least one sensingelement; and extracting information from the scene in response todetection of the frequency pattern.